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This  effort  was  conducted  under  subproject  Z1170-PN.Q3,  Improving  the  Accuracy 
and  Usability  of  Automated  Personnel  Information  Systems.  The  objective  of  this 
subproject  is  to  resolve  fundamental  human  engineering  design  issues  for  Navy  personnel 
office  information  systems  that  will  be  implemented  in  the  mid-1980s.  Candidate 
solutions  to  specific  design  issues  are  to  be  tested  for  application  in  the  user  environment. 

The  objective  of  this  effort  was  to  test  two  methods  of  reducing  the  number  of  errors 
in  the  data  entering  the  Manpower  and  Personnel  Training  Information  System  (MAPT1S) 
and  the  Joint  Uniform  Military  Pay  System  (JUMPS)  from  local  personnel  offices.  Two 
computer  data  entry  systems  were  developed  to  analyze  the  performance  times  and  error 
rates  associated  with  the  use  of  optical  character  recognition  forms. 
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SUMMARY 


Problem 


At  present,  using  optical  character  recognition  (OCR)  forms,  it  takes  at  least  14  days 
to  enter  new  information  into  the  master  data  base  of  the  Navy's  Manpower  and  Personnel 
Training  Information  System  (MAPTIS)  or  the  Joint  Uniform  Military  Pay  System 
(JUMPS).  This  may  stretch  to  90  days  if  the  forms  must  be  returned  to  the  submitting 
office  for  correction  of  errors  (from  10  to  30%  of  the  OCR  forms  contain  significant 
errors).  Although  the  Navy  employs  over  200  people  to  check  and  correct  OCR  forms,  the 
Navy's  personnel  data  base  still  contains  significant  errors  that  lead  to  an  uncertain 
knowledge  of  force  strength,  imprecise  budgetary  and  training  control,  suboptimal 
detailing  of  personnel,  and  delayed  distribution  of  orders. 

Purpose 


The  purpose  of  this  effort  was  to  test  the  effectiveness  of  two  computerized  data 
entry  systems  designed  to  improve  operator  efficiency  and  reduce  the  number  of  errors  in 
Navy  personnel  records. 

Approach  and  Results 

Detailed  data  were  collected  on  operator  performance  times  and  on  the  quantities 
and  types  of  errors  made  using  (1)  a  stand-alone,  self-contained  system  and  (2)  a 
distributed  system  that  used  an  off-site  computer  in  tandem  with  an  on-site  micro¬ 
processor.  Preliminary  analyses  of  performance  times  and  error  rates  were  conducted  to 
assess  user  acceptance  so  that  modifications,  if  needed,  could  be  implemented  before  the 
system  testbed  was  installed  in  the  Point  Loma  Pay/Personnel  Administrative  Support 
System  (PASS)  office  located  in  San  Diego,  California. 

PHASE  1  -  Trial  Use  of  a  Self-Contained  Office  Data  Entry  System 

A  self-contained  data  entry  system  was  installed  in  the  Point  Loma  PASS  office. 
NAVCOMPT  form  3060,  used  for  pay  and  personnel  record  changes,  was  presented  in  a 
query/menu  mode  that  incorporated  extensive  "help"  messages.  The  results  of  the  office 
test  met  with  resistance  from  PASS  office  personnel  and  proved  unsuccessful.  The 
resistance  was  due  largely  to  delays  caused  by  a  long  computer  response  time.  After  a 
short  trial  period,  PASS  office  use  of  the  self-contained  system  was  discontinued.  Entry 
time  comparisons  performed  subsequently  in  a  NAVPERSRANDCEN  laboratory  indicated 
that  it  took  almost  twice  as  long  to  prepare  a  3060  OCR  form  using  the  computer  system 
than  when  using  an  electric  typewriter. 

PHASE  II  -  Preliminary  Analysis  of  a  Distributed  Office  Data  Entry  System 

In  response  to  the  criticism  that  the  response  time  of  the  self-contained  system  was 
"too  long,"  a  faster  "distributed"  office  data  entry  system  incorporating  an  off-site 
minicomputer  was  developed.  NAVPERS  1070/P602R  (Dependency  Application/Record  of 
Emergency  Data)  ("602"  form)  was  chosen  for  testing  the  distributed  system  because  it 
was  rated  by  personnelmen  as  the  most  difficult  and  the  most  time-consuming  form  to 
complete.  The  investigators  believed  that  the  PASS  office  personnel  would  be  more  likely 
to  cooperate  in  the  implementation  of  the  distributed  data  entry  system  if  they  perceived 
that  there  would  be  some  significant  benefit  to  the  PASS  office. 


The  distributed  office  data  entry  system  was  set  up  in  a  NAVPERSRANDCEN 
laboratory.  Subjects  used  the  distributed  system  to  complete  two  602  forms  and  an 
electric  typewriter  to  complete  a  second  set  of  forms.  The  two  sets  of  "personnel  data" 
generated  for  this  test  were  matched  for  number  of  keystrokes  and  placement  of  fingers 
to  simulate  data  that  could  be  found  on  completed  602s  in  any  PASS  office. 

Performance  times  and  error  rates  for  both  the  computerized  and  manual  methods 
were  compared.  Additionally,  the  time  required  to  make  changes  to  a  form  stored  in  the 
computer's  data  base  was  compared  with  the  time  required  to  make  the  same  or  similar 
changes  with  the  electric  typewriter.  Each  operator  completed  an  attitudinal  question¬ 
naire  at  the  end  of  the  testing  session. 

There  was  no  significant  difference  between  the  times  needed  for  initial  completion 
of  the  602  forms  by  either  method,  but  twice  as  many  errors  would  have  entered  the  PASS 
information  system  from  the  manual  task.  The  time  required  to  make  changes  to  a 
personnel  record  already  in  the  computer  data  base  was  one-fourth  the  time  required  to 
make  the  same  change  manually,  since  any  manual  change  necessitated  retyping  the 
complete  form. 

Conclusions 

1.  The  distributed  office  data  entry  system  provided  a  distinct  advantage  over  the 
self-contained  system  in  attaining  the  goal  of  reducing  errors  in  the  information  system 
without  adding  to  the  office  workload. 

2.  PASS  offices  could  substantially  reduce  their  error  rates  by  using  a  system 
similar  to  the  distributed  system  used  in  this  test. 

3.  Using  the  distributed  system  to  make  changes  resulted  in  a  75  percent  time 
savings  over  the  manual  method. 

4.  The  time  saved  in  making  changes  would  be,  by  itself,  a  significant  factor  in 
evaluating  the  cost  effectiveness  of  systems  employing  computer  data  bases. 

Follow-on  Effort 


A  distributed  office  data  entry  system  is  currently  being  installed  in  the  Point  Loma 
PASS  office  to  collect  the  data  required  for  fundamental  research  in  the  design  of 
man/computer  interface  systems.  This  system  will:  (1)  replace  OCR  forms  with 
computer-printed  forms,  (2)  use  a  data  base  that  will  permit  previous  entries  to  be 
recalled  for  modification,  and  (3)  print  summary  reports  for  office  use. 
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INTRODUCTION 


Problem 


At  present,  using  optical  character  recognition  (OCR)  forms,  it  takes  at  least  14  days 
to  enter  new  information  into  the  master  data  base  of  the  Navy's  Manpower  and  Personnel 
Training  Information  System  (MAPTIS)  or  the  Joint  Uniform  Military  Pay  System 
(JUMPS).  This  may  stretch  to  90  days  if  the  forms  must  be  returned  to  the  submitting 
office  for  correction  of  errors  (from  10  to  30%  of  the  OCR  forms  contain  significant 
errors)  (Michna,  Laidlaw,  ic  Obermayer,  1978).  Although  the  Navy  employs  over  200 
people  to  check  and  correct  OCR  forms,  the  Navy's  personnel  data  base  still  contains 
significant  errors  that  lead  to  an  uncertain  knowledge  of  force  strength,  imprecise 
budgetary  and  training  control,  suboptimal  detailing  of  personnel,  and  delayed  distribution 
of  orders. 

Purpose 

The  purpose  of  this  effort  was  to  test  the  effectiveness  of  two  computerized  office 
data  entry  systems  designed  to  improve  operator  efficiency  and  reduce  the  number  of 
errors  entered  into  Navy  personnel  records.  The  first  was  a  self-contained  system  using 
an  on-site  microprocessor  and  the  second  was  a  distributed  system  that  used  an  off-site 
computer  in  tandem  with  the  microprocessor. 

Background 

Data  entry  is  an  essential  part  of  any  large  information  system.  If  the  products  of 
the  data  entry  system  are  inaccurate  or  out-of-date,  the  information  system  is  degraded 
correspondingly.  The  entry  of  pay  and  personnel  data  is  accomplished  in  the  Navy 
primarily  through  the  use  of  manually  typed  OCR  forms.  The  personnel  or  pay 
information  is  typed  in  the  appropriate  boxes  of  specific  forms,  and  the  forms  are  then 
mailed  to  central  computer  facilities  in  Washington,  D.C.  (MAPTIS)  or  Cleveland,  Ohio 
(JUMPS).  At  each  of  these  sites,  the  OCR  forms  are  scanned  by  special  machines  and 
transformed  into  an  electronic  format  suitable  for  further  processing  by  a  large  digital 
computer.  Computer  error  checks  are  made  at  the  scanner  (i.e.,  the  scanner  checks  for 
those  errors  that  can  be  identified  from  other  information  on  the  form),  and  again  at  the 
large  computer  (i.e.,  edits  that  include  checking  against  previously  stored  information 
from  many  sources).  Some  errors  are  corrected  by  error  research  teams  at  the  MAPTIS 
and  JUMPS  sites  but  many  of  the  forms  must  be  returned  to  the  originating  field  office 
for  correction. 

This  process  is  both  slow  and  inaccurate.  Although  full  audits  of  the  personnel  data 
base  are  not  available,  corrections  were  required  in  more  than  80  percent  of  the  records 
examined  in  two  surveys.1’2  In  the  first  survey,  500  interviews  yielded  477  errors.  The 
second  survey  yielded  4792  errors  from  5708  records  (Michna  et  al.,  1978).  These  errors, 
which  found  their  way  into  the  data  base  in  spite  of  herculean  efforts,  are  presumably  an 
indication  of  as  yet  undiscovered  errors  in  the  MAPTIS  and  JUMPS  data  bases. 


‘BUPERS,  Source  data  automation  study  group  (Final  Draft).  Washington,  DC: 
Bureau  of  Naval  Personnel,  24  February  1975. 

2CHNAVPERS  (PERS-53)  Naval  Speedletter,  15  January  1976. 


OCR  errors  detected  by  the  scanners  range  from  10  to  20  percent,  depending  on 
which  one  of  the  approximately  30  forms  is  being  reviewed.  Additional  errors  are  found 
when  the  large  computer  updates  the  data  base  and  when  new  information  is  compared 
with  existing  data  base  information  (Michna  et  al.,  1978,  and  footnotes  1  and  2). 

Table  1  lists  the  types  of  errors  found  on  two  OCR  forms.  Form  3060  is  for  pay 
purposes  and  Form  610  is  for  personnel  information  changes.  Both  forms  occur  in  high 
volume  and  many  types  of  errors  are  detected  and  corrected  manually;  however,  only  the 
primary  types  of  errors  that  are  applicable  to  this  report  are  shown.  The  error  data 
presented  in  Table  1  were  collected  in  1977,  before  the  Navy's  consolidation  of  field  pay 
and  personnel  offices. 


Table  1 

OCR  Form  Errors 


Error  Type 

Percent  of  Total  Errors 

Form  3060a 

Form  610^ 

Undecipherable  entry 

25.0 

40.0 

No  cross-check 

12.3 

7.7 

Characters  in  error 

5.6 

16.2 

Entry  missing 

5.4 

21.3 

Invalid  date 

8.9 

13.4 

Invalid  amount 

5.1 

NA 

Name/SSN  mismatch 

6.7 

NA 

Total 

69.0 

98.6 

aMilitary  Pay  Order  (NAVCOMPT  3060). 

^Record  of  Personnel  Actions  (NAVPERS  1710/610). 


Prior  to  1979,  there  were  more  than  3000  Navy  field  units  submitting  OCR 
documents.  It  was  clearly  not  feasible  to  automate  so  many  small  offices,  especially 
since  half  of  the  field  units  submitted  less  than  1000  documents  per  year,  and  half  of 
these  submitted  less  than  500  per  year.  With  the  advent  of  the  Pay/Personnel 
Administrative  Support  System  (PASS),  which  integrated  pay  and  personnel  functions,  the 
number  of  submitting  offices  was  reduced  to  approximately  500. 


PHASE  I— TRIAL  USE  OF  A  SELF-CONTAINED  OFFICE  DATA  ENTRY  SYSTEM 
Purpose 

The  purpose  of  Phase  1  was  to  test  the  effectiveness  of  using  a  self-contained 
computer  configuration  for  detecting  errors  at  PASS  field  offices  as  data  are  being 
entered  onto  the  OCR  form. 
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A  self-contained,  computerized,  office  data  entry  system  was  designed  in  a  prelimi¬ 
nary  effort  to  collect  detailed  data  on  the  quantities  and  types  of  errors  made  in  local 
PASS  offices-  The  self-contained  system  was  tested  both  in  a  PASS  office  and  in  a 
NAVPERSRANDCEN  laboratory.  The  system  had  a  capability  for  automatic  error 
detection,  provision  for  manual  corrections,  and  a  printer  for  processing  the  OCR  form. 
The  form  used  for  this  test  was  the  one  used  for  military  pay  actions,  NAVCO.V1PT  3060 
(see  Figure  1).  The  self-contained  system  also  automatically  recorded  errors  for  later 
analysis. 

The  software,  written  in  BASIC,  presents  NAVCOMPT  3060  in  a  query /menu 
mode.  The  numbered  "blocks"  of  the  form  are  presented  as  two  series  on  the  CRT  display 
screen.  The  first  series  consists  of  information  specific  to  the  originating  office,  and  the 
second,  information  specific  to  the  individual  service  member.  The  computer  flow 
diagram  is  shown  in  Figure  2.  A  hierarchial  system  of  prompts,  cues,  and  detailed 
explanations  allowed  the  novice  to  use  the  system  as  well  as  the  highly  trained 
personnelman. 

The  "reason  for  change"  block  is  the  main  source  of  errors,  and  software  was 
developed  in  a  menu  format  to  help  in  the  selection  of  the  proper  entry  code.  Cues,  helps, 
and  project  aids  allowed  complex  abbreviations  to  be  selected  from  a  menu-type  list  on 
the  CRT,  and  the  correct  abbreviations  or  codes  were  entered  automatically.  The  menu 
format  eliminated  errors  due  to  spelling  or  selection  of  inappropriate  codes.  Information 
was  transcribed  onto  the  3060  OCR  form  through  use  of  a  Diablo  printer  equipped  with  an 
OCR  font.  The  capability  for  repeatedly  printing  the  completed  form  at  the  time  of  data 
entry,  without  reentering  the  information,  was  made  available. 

The  self-contained  system  also  included  the  capability  for  counting  entries  to  enure 
that  all  necessary  entries,  but  no  extra  ones,  were  made.  Edits  were  designed  to  scan  the 
OCR  form  for  appropriate  alpha,  numeric,  and  special  characters,  and  for  the  total 
number  of  characters,  among  other  features.  Errors  detected  by  the  computer  edits, 
including  undecipherable  entries,  were  presented  to  the  data  entry  operator  as  they  were 
detected  so  that  corrections  could  be  made  on  the  CRT  before  the  form  was  printed. 
These  errors  accounted  for  two-thirds  of  the  total  errors  found.  The  errors  were 
displayed  as  shown  in  Figure  3  and  recorded  for  subsequent  statistical  analysis. 

Results  and  Conclusions 

The  tests  conducted  in  the  PASS  office  met  with  resistance  from  PASS  office 
personnel,  due  mainly  to  a  long  computer  response  time.  PASS  personnel  determined  it 
was  easier  and  quicker  to  prepare  the  3060  form  on  a  standard  electric  typewriter.  At  the 
PASS  supervisory  level,  it  was  determined  that  the  error  reduction  factor  did  not 
compensate  for  the  lost  operator  time;  consequently,  the  local  office  perceived  no 
benefit.  The  office  test  was  therefore  terminated  early  and  the  system  was  moved  to  a 
NAVPERSRANDCEN  laboratory  for  further  tests. 

Four  subjects  used  the  self-contained  system  and  standard  electric  typewriters  to 
prepare  NAVCOMPT  3060  forms  under  controlled  conditions  in  a  NAVPERSRANDCEN 
laboratory.  All  of  the  subjects  were  experienced  in  the  preparation  of  OCR  forms  using 
an  electric  typewriter.  One  subject  was  tested  first  after  a  short  orientation  period  on 
the  self-contained  system,  and  again  after  several  weeks'  experience  with  the  system. 
The  subjects'  experience  on  the  computer  system  was  labeled  as  either  "inexperienced" 
(less  than  1  week)  or  "experienced"  (6-9  weeks). 


Figure  2.  Computer  flow  diagram  for  processing  NAVCOMPT  3060  on  the  self-contained  system 


Message 

Applicable 

Block  Numbers 

_ 

00007  TOO  FEW  CHARACTERS 

1,  5,  7,  9 

00020  TOO  MANY  CHARACTERS 

1 

00017  INCORRECT  CHARACTERS 

1,  2,  A,  5,  6,  7,  8,  9,  10 

00006  BLANK  FIELD 

1,  2,  3,  A,  9,  10,  11,  68 

00016  INVALID  DATE 

2,  6,  8 

DATE  OF  THIS  TRANSACTION  MUST  BE  LESS 

6,  8 

THAN  OR  EQUAL  TO  TODAY'S  DATE 

ADDITIONAL  HELP  MESSAGES 

BLOCK  7  IS  BLANK.  IS  IT  SIGNIFICANT  TO 

7 

PAY  ACTION  OR  IS  THERE  A  DATE  TO  BE  PUT 

IN  BLOCK  8? 

BLOCK  10  IS  BLANK.  IS  THERE  TO  BE  A 

10 

MONETARY  AMOUNT  FOR  THIS  PERSON? 

Figure  3.  Error  messages  displayed  by  self-contained  system  (typical). 


Overall,  the  individual  and  total  time  comparisons  indicated  that  using  the  self- 
contained  system  required  more  time  than  using  the  standard  electric  typewriter.  Where 
only  one  transaction  for  a  single  service  member  was  to  be  recorded,  data  could  be 
entered  in  approximately  50  percent  less  time  when  using  a  standard  typewriter  than  when 
using  the  self-contained  computer  system.  Even  forms  requiring  multiple  entries  were 
produced  faster  (80%  of  the  computer  time)  when  using  a  standard  typewriter.  The  time 
required  for  the  manual  task  did  not,  however,  include  any  of  the  normal  editing  time, 
either  by  the  personnelmen  or  by  the  supervisor.  In  addition  to  the  entry  time,  the 
operator  using  the  self-contained  system  was  required  to  display  the  completed  form  on  a 
CRT  for  editing  and  then  print  the  final  form.  The  error  checking  features  were  not 
considered  to  be  an  offsetting  factor  by  the  test  subjects. 

A  major  factor  in  user  acceptance  of  any  new  system  is  that  the  user  must  perceive  a 
distinct  advantage  in  using  the  new  system  (the  computer  system  in  this  case)  over  the 
current  system  (filling  out  forms  manually  using  an  electric  typewriter).  A  reduction  of 


the  office  workload,  which  should  be  a  primary  consideration  of  any  computer  configura¬ 
tion,  was  not  effected  by  the  self-contained  system. 


PHASE  0— PRELIMINARY  ANALYSIS  OF  A  DISTRIBUTED  DATA  ENTRY  SYSTEM 
Purpose 

The  purpose  of  Phase  II  was  to  develop  and  test  the  effectiveness  of  a  faster  and 
more  efficient  office  data  entry  system  employing  distributed  data  processing  capabili¬ 
ties.  This  system  incorporated  an  off -site  minicomputer  for  detecting  errors  at  the  field 
offices  as  data  were  entered  into  a  master  information  system.  For  a  preliminary 
assessment,  the  distributed  data  entry  system  was  tested  in  a  laboratory  at  NAVPERS- 
RANDCEN  to  establish  performance  times  and  error  rates.  After  completion  of  the 
preliminary  assessment,  the  distributed  system  was  removed  from  NAVPERSRANDCEN 
and  installed  in  the  Point  Loma  PASS  office,  where  additional  research  will  be  performed. 

Approach 

The  Distributed  System  Testbed 

The  distributed  office  data  entry  system  testbed  was  an  expanded  reconfiguration  of 
the  self-contained  system.  New  software  was  designed  so  that  the  entries  or  "blocks"  on 
the  OCR  form  were  individually  displayed  on  the  CRT  screen  in  a  manner  similar  to  the 
display  shown  on  the  self-contained  system.  If  an  incorrectly  formatted  entry  was  made, 
an  error  message  was  displayed  that  was  specific  to  that  block.  Erasures  were  made  by 
backspacing,  and  the  operator  was  not  allowed  to  proceed  until  the  error  had  been 
corrected. 

An  expanded  edit  mode  provided  capabilities  for  displaying  the  recorded  data  and  for 
correcting  any  errors  that  might  be  found  without  reentering  the  complete  form.  Only 
the  keystrokes  required  for  the  specified  change  were  necessary  to  update  the  data  base. 

One  of  the  criticisms  of  the  self-contained  system  was  the  "long"  computer  response 
time.  Williges  (1980)  reported  that  operators  will  tolerate  a  maximum  system  delay  of  2 
to  4  seconds  before  experiencing  irritation  and  work  degradation.  In  response  to  this 
criticism,  a  PDP  11/45  minicomputer  was  included  in  the  distributed  configuration.  The 
addition  of  increased  memory  assured  faster  response  time  and  the  ability  to  include  a 
data  base  that  could  provide  many  side  benefits  to  the  user. 

The  Dependency  Application/Record  of  Emergency  Data  (NAVPERS  1070/P602R) 
form  was  singled  out  as  the  most  difficult  and  time  consuming  form  to  complete.  The  602 
is  used  as  a  legal  document  with  the  service  member's  signature  attesting  to  claims  of 
dependency  and  to  disposition  of  assets  in  case  of  death;  every  service  member  is  required 
to  have  a  602  form  in  his  or  her  personnel  jacket. 

The  difficulty  with  the  602  stems  partially  from  its  complexity.  It  is  two  records 
combined  into  one  and  consists  of  80  entry  blocks  (see  Figure  4).  There  are  five  copies  to 
each  form.  The  first  three  are  sent  to  Cleveland,  the  fourth  to  Washington,  and  the  last 
copy  is  retained  in  the  service  member's  folder.  However,  if  there  are  no  changes  in 
dependencies,  only  the  last  two  copies  are  filled  out.  Further,  to  add  to  the  problem,  the 
backs  of  Parts  I  and  II  of  the  form  are  different.  Precisely  because  of  these  difficulties 
and  because  of  the  time  involved  in  filling  it  out,  the  602  was  chosen  as  the  experimental 
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4.  Dependency  Application/Record  of  Emergency  Data  Form,  front  page  of  Part 
II  (NAVPERS  1070/P602R). 


form.  It  was  believed  that  the  PASS  office  would  be  more  willing  to  cooperate  in  the 
implementation  of  the  distributed  data  entry  system  if  it  could  be  demonstrated  that  the 
office  would  reap  a  significant  benefit  from  the  faster  handling  of  this  troublesome  form. 

An  immediate  advantage  was  seen  with  the  addition  of  a  data  base  that  would  allow 
the  personnelman  to  reprint  the  form,  with  changes,  without  manually  reentering  all  of 
the  data  on  the  form.  Using  the  manual  method  of  preparation,  the  form  must  be 
completely  retyped  each  time  a  change  is  required.  When  an  error  occurs,  it  must  be 
over-typed  with  the  OCR  "tree  block."  If  an  uncorrectable  error  occurs  (where  the 
correct  information  will  no  longer  fit  in  the  prescribed  block),  the  form  must  be  discarded 
and  a  new  form  retyped. 

With  the  addition  of  the  data  base,  the  number  of  keystrokes  required  for  initial 
completion  of  the  form  remained  the  same  but,  as  changes  became  necessary,  only  the 
keystrokes  pertinent  to  the  changes  would  be  necessary.  Additionally,  the  data  base 
would  be  available  for  the  printing  of  summary  reports  and  audit  trials  for  other  office 
needs. 

Procedure 


The  participants  in  the  laboratory  test  were  seven  males  and  three  females  who  were 
randomly  selected  from  a  pool  of  Navy  personnelmen  (PN  rating)  from  the  Point  Loma 
PASS  office  and  Navy  civilian  clerical  personnel  from  NAVPERSRANDCEN.  The  mean 
age  was  34.8  years.  All  of  the  participants  were  experienced  typists  familiar  with  Navy 
procedures  and  the  general  format  of  Navy  OCR  forms. 

The  participants  were  given  brief  written  instructions  that  explained  the  experiment. 
They  were  encouraged  to  ask  questions  at  any  time  to  clarify  their  role  in  the  experiment 
and  its  specific  intent.  The  participants  were  told  that  the  current  manual  system  was 
being  compared  to  the  proposed  microcomputer  system.  It  was  emphasized  that  the 
participants  themselves  were  not  being  tested  but,  rather,  that  the  purpose  was  to 
evaluate  the  ease  of  understanding  and  operating  the  two  systems  and  to  assess  timing 
issues  involved  in  each  system. 

A  user's  manual,  designed  to  serve  as  a  training  guide  and  job  aid,  was  given  to  each 
subject.  A  researcher  reviewed  the  manual  with  the  subject  just  prior  to  the  start  of  each 
computer-assisted  task.  A  flowchart  in  the  user's  manual  delineated  the  operator's 
procedures  and  decision  points  (see  Figure  5).  The  manual  described  in  detail  the 
procedure  for  filling  out  the  602  form  for  each  of  the  computer  conditions.  Using  the 
manual,  the  operator  could  see  what  would  be  displayed  on  the  CRT  during  initial  entry  of 
a  form,  how  to  display  data,  how  to  make  corrections,  and  how  to  print  the  data  on  an 
OCR  form  or  transmit  the  data  on-line  to  a  remote  point.  The  operator  could  see  in  a 
block-by-block  printed  format  the  exact  sequence  of  events  that  would  occur.  Addition¬ 
ally,  the  user's  manual  contained  detailed  explanations  for  those  blocks  that  had  specific 
restrictions  (see  Figure  6).  The  operators  were  allowed  as  much  time  as  necessary  to  read 
and  understand  the  computer  operation.  The  mean  time  spent  on  the  user's  manual  was 
17.7  minutes.  The  operators  were  asked  to  enter  two  practice  forms  on  the  computer 
before  beginning  the  experimental  forms.  Mean  time  spent  in  the  practice  session  was 
44.2  minutes. 

The  computer-assisted  task  consisted  of  subjects  filling  out  two  separate  602  forms 
on  the  computer  (80  blocks  on  each  form).  After  the  forms  were  completed,  subjects 
were  asked  to  make  two  changes  on  a  602  form  that  was  already  on  file  (in  the  data  base). 
The  spouse's  citizenship  was  to  be  changed  to  indicate  naturalized  citizenship  and  the 
"effective  date"  was  to  be  changed  to  reflect  this  action.  This  meant  that  the  subject  had 
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Figure  5.  Distributed  data  entry  system  operations  flowchart  for  processing  the 
NAVPERS  1070/P602R  form  on  the  distributed  system. 
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When  a  social  security  number  is  r.ot  found  in  either  tiie  Officer  or  Enlisted 
data  base  file,  and  this  information  is  verified  by  the  operator,  the  602 
(Page  2)  messages  will  appear  biock  by  block  in  the  following  order: 


Figure  *.  Excerpt  from  the  distributed  offlc.  «-«»  traiuiug/ruf eruuce  .usual 


to  call  for  a  file  from  the  data  base  (by  entering  the  social  security  number  and  rank  of 
the  individual),  verify  that  the  correct  file  was  displayed,  go  into  the  Change  mode  to 
make  changes  in  two  specific  blocks,  and  print  the  corrected  form.  Entry  time  for  a 
completed  task  was  measured  on  the  computer  from  the  second  that  the  operator  chose 
the  form  to  the  time  that  the  form  was  printed.  A  breakdown  of  the  time  spent  in  the 
Change,  Display,  and  Print  modes  was  available  for  each  operator.  No  overt  timing 
procedures  were  needed  since  the  computer  recorded  all  timing  and  error  information. 

The  manual  task  consisted  of  the  subjects  typing  two  forms  on  an  IBM  Selectric 
typewriter  following  normal  office  procedures.  The  subjects  were  required  to  fill  in 
blocks  1-46  and  67-76  on  the  front  page  for  all  five  copies,  remove  the  form  from  the 
typewriter,  reinsert  copies  4-5,  and  fill  in  blocks  47-66  (see  Figure  3).  The  backs  of  Part  I 
and  Part  II  had  to  be  filled  out  separately  (turning  them  over  and  reversing  the  carbons) 
since  they  contained  different  types  of  information.  A  stopwatch  accurate  to  the  tenth  of 
a  second  was  used  to  time  the  subjects  on  each  form  starting  with  form  insertion  into  the 
typewriter  and  ending  when  the  finished  form  was  handed  to  the  experimenter. 

Errors  were  separated  into  the  two  ca+egories  of  "detected  and  corrected"  errors  and 
"undetected"  errors  (i.e.,  those  that  wou*i  gone  into  the  information  system).  The  OCR 
"tree  block"  errors  was  used  as  the>  measure  of  detected  errors.  Errors  detected 

on  the  computer-assisted  task  we*  *  iUhuf-r  tsv.  and  registered  as  error  messages,  backspace 
counts,  and  corrections  made  to  a  fp'-v  t  c  ‘  iock  in  the  Change  mode.  Undetected  errors 
on  both  the  computer  and  manual  '  jsf  s  were  measured  by  discrepancies  between  the 
subjects'  completed  forms  and  the  orig.iy>t  experimental  forms.  "Detected  and  corrected" 
and  "undetected"  errors  were  a'KVyz*'-'!  Detween  the  manual  and  the  computer  system 
using  t  tests  with  correlated  m a. 

Two  equivalent  sets  o*  personnel  data"  were  designed  for  each  experimental 
condition  and  designated  1.1  and  1.2  for  the  computer  task  and  2.1  and  2.2  for  the  manual 
task.  The  sets  replicated  data  that  might  be  found  on  a  602  form  in  any  service  member's 
file  and  were  matched  for  number  of  keystrokes  and  placement  of  fingers.  Sets  1.1  and 
2.1  had  685  keystrokes;  sets  1.2  and  2.2  had  644  keystrokes.  The  difference  in  keystrokes 
reflected  the  addition  of  two  dependent  children  to  sets  1.1  and  2.1. 

To  eliminate  order  effects,  the  subjects  were  randomly  divided  and  half  performed 
the  computer -assisted  task  first  and  half  performed  the  manual  task  first.  At  the  end  of 
the  experimental  session,  each  subject  was  asked  to  fill  out  a  biographic/attitudinal 
questionnaire. 


Computer  Hardware.  The  distributed  system  used  a  Futuredata  Microsystem 
microprocessor  consisting  of  four  modules:  (l)a  mainframe  (8080  CPU,  65K  bytes),  (2) 
dual  floppy  disks,  (3)  a  standard  typewriter  keyboard,  and  (4)  a  CRT  video  display. 

Attached  to  this  system  was  a  Diablo  Hyterm  1620  on-line  printer.  The 
microprocessor  was  connected,  via  telephone,  to  the  PDP  11/45  minicomputer  that  was 
used  later  to  store  the  data  base  for  the  PASS  office. 

Software.  The  software  presented  the  602  form  in  a  block-by-block  query/menu 
mode  on  the  CRT.  Edits  for  errors  with  messages  appropriate  to  the  specific  block  were 
included.  A  microsystem  BASIC  Editor,  Monitor,  and  Utility,  and  a  Time/Error  file  were 
also  included  in  the  software.  The  Editor  provided  the  capability  of  creating  or  updating 
any  text  files;  the  Monitor  created  or  deleted  system  files;  the  Time/Error  file  presented 


a  block-by-block  listing  of  the  selected  program  and  identified  system  time,  operator 
time,  and  types  of  errors  (i.e.,  number  of  backspaces  used,  error  message  numbers,  and 
the  number  of  times  that  the  Change  mode  was  used  to  correct  information  in  a  specific 
block). 

Manual  Typewriter.  Manual  entry  of  the  forms  was  accomplished  using  an  IBM 
Selectric  typewriter  with  an  OCR  font. 

Results  and  Conclusions 


A  between-within-within  (2x2x2)  analysis  of  variance  with  repeated  measures  deter¬ 
mined  there  was  no  significant  difference  between  the  times  required  to  complete  the 
forms  on  the  computer  (14.6  minutes)  or  to  do  them  manually  (16.5  minutes),  F (1,  24)  = 
1.88.  All  tests  were  conducted  at  the  2  <  *05  level  of  significance.  Significance  was 
found  only  between  the  first  and  second  forms  in  both  the  computer-assisted  and  manual 
tasks,  F(l,  24)  =  9.57  (see  Table  2). 


Table  2 

Times  Required  for  Computer  and  Manual  Tasks 


Time  by  Task  per  Form  602  (in  Minutes) 


Timing  Measurement 

Completion 
of  Form 
on  Computer 

Making  Two 
Changes  to  a 
Form  in  File 
on  Computer 

Manually  Com¬ 
pleting  Form 
or  Making  Two 
Changesa 

Entry  <Sc  Printing  Times 

Mean 

14.56 

4.13 

16.46 

Median 

14.20 

4.11 

14.60 

Mode 

11.18,17.50 

4.11 

10.38 

Variance 

8.75 

2.34 

42.26 

Standard  Deviation 

2.96 

.58 

6.50 

Mean  Entry  Time 

12.20 

2.20 

— 

Mean  Printing  Time 

2.40 

1.89 

— 

aSince  making  changes  manually  necessitated  retyping  the  entire  form,  the  times  for 
initial  preparation  of  form  or  for  making  changes  would  be  the  same. 


The  number  of  "undetected"  errors  that  would  have  entered  the  Navy's  master 
information  system  was  significantly  greater  for  forms  filled  out  manually  than  for  forms 
completed  on  the  distributed  system,  tf9)  =  2.36.  That  is,  when  operators  filled  out  the 
forms  manually,  30  percent  of  the- errors  (1.3  errors  per  form)  would  have  been 
incorporated  into  the  PASS  information  system,  whereas  forms  entered  on  the  computer 
yielded  a  6  percent  "undetected"  error  rate  (0.55  errors  per  form)  (see  Table  3). 


Table  3 


Error  Rates  for  Computer  and  Manual  Tasks 


Mean  Errors  per  Form  602  by  Task 


Error  Detection /Correction 
Categories 

Completion 
of  Form  on 
Computer 

Making  Two 
Changes  to  a 
Form  in  File 
on  Computer 

Manually  Com¬ 
pleting  form 
or  Making  Two 
Changes3 

Operator  detected/corrected 

8.25 

0.10 

3.00 

(%  of  total  errors) 

(87) 

(33) 

(70) 

Computer  detected;  operator 

0.70 

0.20 

— 

corrected  (%  of  total  errors) 

(7) 

(67) 

Total  detected  errors 

8.95 

.30 

3.00 

(%  of  total  errors) 

(94) 

(100) 

(70) 

Undetected  errors0 

0.55 

0.00 

1.30 

(%  of  total  errors) 

(6) 

(0) 

(30) 

Total  errors 

9.50 

0.30 

4.30 

aSince  making  changes  manually  necessitated  retyping  the  entire  form,  the  times  for 
initial  preparation  or  for  making  changes  would  be  the  same. 

^Errors  that  would  enter  the  information  system. 


Operators  detected  and  corrected  significantly  more  of  their  errors  when  using  the 
distributed  system  than  they  did  when  preparing  the  forms  manually,  _t(9)  =  5.00.  Forms 
entered  on  the  computer  had  a  "detected  and  corrected"  error  rate  of  94  percent  (8.95 
errors  per  form),  while  those  prepared  manually  had  a  70  percent  "detected  and 
corrected"  error  rate  (3.0  errors  per  form). 

The  mean  time  required  to  make  two  changes  to  a  record  already  in  the  computer 
data  base  (4.1  minutes)  was  one-fourth  of  the  mean  time  required  for  the  manual  task 
(16.5  minutes).  This  is  an  example  where  only  two  changes  were  made  to  the  form  (the 
date  of  the  change  plus  the  change);  however,  making  the  same  two  changes  manually 
would  have  necessitated  retyping  the  complete  form.  The  "detected  and  corrected"  errc- 
rate  for  forms  already  in  file  was  .05  errors  per  form.  There  were  no  "undetected"  errors 
that  would  have  entered  the  information  system. 

There  was  no  significant  difference  between  the  times  needed  to  complete  a  form 
initially  in  the  computer-assisted  task  or  the  manual  task;  however,  twice  as  many  errors 
would  have  entered  the  PASS  information  system  from  the  manual  task.  Changes  made  to 
the  602  form  using  the  computer  Change  mode  resulted  in  a  75-percent  time  savings  ov_r 
the  manual  method. 
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On  the  attitud'inal  questionnaire  (Figure  7),  all  operators  indicated  that  they 
preferred  the  computer  system.  Reasons  given  for  this  preference  were  that  (1)  it  was 
easier  to  find  and  correct  mistakes,  (2)  the  form  did  not  have  to  be  retyped,  and  (3)  it  was 
faster  and  cleaner  (no  carbon  copies  to  "mess  with").  All  of  the  subjects  recommended 
elimination  of  the  OCR  form,  preferring  on-line  transmission  of  the  data.  The  prompt 
messages  and  edits  were  categorized  as  "too  easy"  by  40  percent  of  the  operators  and  it 
was  suggested  that  the  prompts  could  be  streamlined  with  fewer  "help"  explanations.  This 
would  further  decrease  the  computer  entry  time.  As  this  effort  was  a  preliminary 
analysis  using  a  small  N  (10),  caution  should  be  used  in  generalizing  the  results  beyond  the 
sample. 


602  DEPENDENCY  FORM  (PAGE  2) 

Experimental  Questionnaire 

1.  Your  initials:  _  Date: _ 

2.  Grade  or  Rank: _ 

3.  Age:  _ 

4.  Sex:  M  F  (Circle  one) 

5.  How  many  words  per  minute  do  you  type? _ 

6.  Did  you  prefer  the  manual  typewriter  or  the  automated  system? 

Manual  Automated  (Circle  one) 

7.  Did  using  the  Microsystem  make  using  this  form  (P602R)  any  easier? 

Harder  Same  Easier  (Circle  one) 

Why? 

8.  What  do  you  think  about  the  microsystem  generally? 

9.  What  features  about  the  system  do  you  like? 

10.  Did  the  Users  Manual  and  the  Edits  in  the  program  give  you  enough  information? 

11.  Are  the  edits  and  prompts 

too  easy?  good?  too  hard?  (Circle  one) 

12.  Do  you  have  any  other  observations  regarding  the  system? 

Figure  7.  Attitudinal  questionnaire. 
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DISCUSSION 


The  distributed  system  provided  a  distinct  advantage  over  the  self-contained  system 
in  reducing  errors  in  the  information  system  without  adding  to  the  office  workload.  Based 
on  observation  of  the  short  time  (one  practice  session)  that  it  took  the  subjects  to  become 
familiar  with  the  computer,  it  is  estimated  that  entry  of  a  complete  form  without  errors 
could  be  accomplished  in  8  minutes  or  less  by  an  experienced  operator. 

Although  fewer  errors  would  have  entered  the  PASS  information  system  using  the 
computer,  users  produced  more  errors  during  initial  entry  of  the  data.  This  phenomenon 
was  also  noted  by  Fields,  Maisano,  and  Marshall  (1978)  in  an  experiment  where  subjects 
typed  text  into  a  computer  system.  Their  system  included  both  a  spelling  correction 
feature  (comparing  unknown  terms  with  a  list  of  correctly  spelled  terms)  and  an 
autocompletion  feature  (using  initial  parts  of  words  or  initial  parts  of  established 
abbreviations).  User  error  rates  increased  when  the  autocompletion  feature  was  avail¬ 
able.  Fields  et  al.  believe  that,  while  inexperienced  users  show  a  strong  preference  for 
automatic  correction  options,  they  initially  experience  difficulty  in  using  them  correctly. 

This  could  partially  explain  the  increased  error  rates  when  using  the  computer  system 
in  this  analysis.  Another  explanation,  which  would  be  more  difficult  to  test,  could  be  that, 
since  users  know  error  correction  features  are  available  and  final  products  will  be  error- 
free,  they  experience  a  greater  sense  of  freedom  when  using  the  computer  system.  This 
combination  of  "inexperience"  and  "sense  of  freedom"  seems  to  contribute  to  increased 
error  rates  for  inexperienced  users. 

In  a  service  member's  career,  there  are  often  changes  of  address  and  in  dependencies, 
all  of  which  currently  necessitate  retyping  the  entire  602  form.  A  75-percent  time 
savings  in  recording  these  changes  would  make  computerization  a  significant  factor  in 
improving  cost  effectiveness.  Since  the  mean  printing  time  per  form  is  2.0  minutes,  a 
remote  printing  procedure  could  be  established  as  long  as  hard  copy  forms  are  required  for 
legal  purposes.  This  would  free  additional  time  for  the  operators,  enabling  them  to 
proceed  with  other  work  tasks.  PASS  offices  have  expressed  the  need  for  management 
information  and  word  processing  capabilities,  in  addition  to  improved  data  entry.  User 
acceptance  is  likely  to  be  a  problem  if  piecemeal  automation  increases  office  workload. 


CONCLUSIONS 

1.  The  distributed  office  data  entry  system  provided  a  distinct  advantage  over  the 
self-contained  system  in  attaining  the  goal  of  reducing  errors  in  the  information  system 
without  adding  to  the  office  workload. 

2.  PASS  offices  could  substantially  reduce  their  error  rates  by  using  a  system 
similar  to  the  distributed  system  used  in  this  test. 

3.  Using  the  distributed  system  to  make  changes  resulted  in  a  75  percent  time 
savings  over  the  manual  method. 

6.  The  time  saved  in  making  changes  would  be,  by  itself,  a  significant  factor  in 
evaluating  the  cost  effectiveness  of  systems  employing  computer  data  bases. 


FOLLOW-ON  EFFORT 


A  distributed  office  data  entry  system  is  currently  being  installed  in  the  Point  Loma 
PASS  Office  to  collect  the  data  required  for  fundamental  research  in  the  design  of 
man/comf  ter  interface  systems.  This  system  will: 

1.  Replace  the  OCR  with  computer-printed  forms.  It  is  easier  and  faster  to  make 
corrections  when  a  computer  provides  immediate  feedback  to  the  operator;  consequently, 
the  number  of  errors  that  enter  the  information  system  would  be  substantially  reduced. 

2.  Use  a  data  base  that  will  permit  previous  entries  to  be  recalled  for  modification. 
Time  saved  in  entering  and  retrieving  data  from  the  master  information  system  will 
reduce  manpower  costs  (e.g.,  personnel  could  use  the  extra  time  to  perform  other  office 
tasks).  Additionally,  once  the  information  is  correctly  entered,  the  likelihood  of  creating 
new  errors  on  previous  data  would  be  greatly  reduced. 

3.  Print  summary  reports  for  office  use.  Audit  trails  for  specific  personnel,  or 
summary  reports  of  personnel  actions,  could  be  incorporated  into  the  system  to  provide 
management  with  a  convenient  and  useful  management  tool. 
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